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Executive Summary 


This study reports on results of a survey of the Council of State Science Supervisors (CSSS). CSSS. 
is a professional organization composed of supervisors of science education in State Education 
Agencies. The organization works to sustain and nurture a dynamic learning community that 
empowers its members to be effective and articulate advocates for quality science education at 
the local, state, and national levels. The National Center for Research in Policy and Practice, a 
center funded by the Institute of Education Sciences at the U.S. Department of Education, 
conducted the study. 


The survey focused on the roles and activities that CSSS members take on and participate in as 
supervisors of science education, as well as on how they use research to inform state decisions 
related to implementation of the Framework for K-12 Science Education.’ While State Education 
Agencies play key roles in supporting the implementation of standards that align with the 
Framework, with state science supervisors taking a lead role, little is known about how these 
individuals use research to inform their decisions. 


The Council of State Science Supervisors includes both current and past state leaders, and this 
survey study included both these groups. A total of 61 individuals responded to the survey from 
38 states. The overall response rate for the survey was 62%. On average, survey participants 
had five years of experience as state science leaders, and the majority of the sample was white 
and female. 


The survey used items that were previously developed and validated via a national survey of 
research use conducted by the National Center for Research in Policy and Practice." 
‘Additionally, the survey team drew on prior studies" to author social network questions related 
to CSSS members’ research networks. Finally, survey items pertaining to CSSS members’ roles. 
and activities were developed in collaboration with CSSS members. 


Definition of Research Used in the Study 

For the purposes of this study, we defined “research” as an activity in which people employ 
systematic, empirical methods to answer a specific question. In this sense, research is different 
than the practice of looking at data from the district, school, or classroom, which is more open- 
ended and seldom addresses specific research questions. 


Roles and Activities 

The survey asked respondents which roles they have assumed within the CSSS organization. 
CSSS members most often reported participating in their state science conference and as ad- 
hoc committee contributors or participants. They reported serving as CSSS officers least often, 
as well as organizers of their state science fairs. 


Respondents were also asked to report how often they had participated in various CSSS 
activities in the last three years. Of the activities included on the survey, respondents reported 
reading information on the CSSS listserv most frequently, as well as consulting with other CSSS 


members. Respondents were less likely to report visiting other states or attending CSSS 
meetings, such as the annual meeting, Board meetings, and committee meetings. 


In addition to CSSS activities, survey respondents reported the kinds of state-level activities 
they engaged in over the last year. Of these state activities, respondents reported being most 
involved with leading state science standards development and professional development 
design, whereas they were less involved in curriculum selection or adoption and the allocation 
of federal or grant funds. 


CSSS Activities that Research Supports 

In addition to reporting the types of activities they engaged in as CSSS members, respondents 
also indicated the extent to which they learned about research or research-based tools via 
these activities. The majority of respondents who attended CSSS Annual Meetings and BCSSE 
Meetings, as well as workshops or talks by researchers, reported learning about research 
findings and research-based tools through these activities. Conversely, the fewest number of 
respondents reported learning about research while visiting other states or attending CSSS 
board meetings. 


Specific Pieces of Research CSSS Members Found Useful or Shared with Others 
The survey asked CSSS members to name a specific piece of research they found useful for 
informing their state’s decisions related to the implementation of the Framework, as well as a 
piece of research they shared with district or school leaders. Respondents reported using and 
sharing research that focused on instructional practices or student learning and development in 
science, yet few mentioned using or sharing research pertaining to particular student 
subgroups (e.g., African American students, English learners). The pieces of research 
respondents named were often research reports or policy briefs, and many of the reports were 
published via the National Research Council. 


Sources Leaders Used to Obtain Research 


The survey listed 13 different sources where CSSS members might obtain research. Of these, 
many respondents indicated seeking out research through CSSS or the National Science 
Teachers Association (NSTA), from other state colleagues, and directly from university 
researchers. Far fewer respondents indicated seeking out research from Regional Educational 
Laboratories (RELs) or the National Science Education Leadership Association (NSELA), the 
professional association for district leaders in science. 


Effort to Acquire Research 

We asked CSSS members to indicate whether they would seek out research under different 
conditions. Although a majority said they would look for research to inform a new problem or 
decision, fewer said they would contact researchers directly under these circumstances, 
especially researchers they do not already know. 


Research Networks 

The survey asked CSSS members to whom they have turned for research to inform their state’s 
implementation of the Framework. While respondents could list up to 12 individuals, an 
average of six individuals were named. Of those, respondents turned to an average of 3 
individuals for research related to assessment and curriculum, and an average of 4 individuals 
for research related to instruction and professional development. 


In total, respondents listed 144 different individuals, including 16 current CSSS members, 6 
former CSSS members, 75 university researchers, 29 individuals working for non-profit 
organizations, 8 members of state departments of education, and 7 local educational leaders. 
Although there was variability in how often each individual was named by respondents, former 
CSSS members were named most frequently by an average of 7 respondents. 


Introduction 


The core studies conducted by the National Center for Research in Policy and Practice focus on 
research use among school and district leaders, but leaders in state education agencies also 
make use of research to inform their decision making. A recent study of research use among 
state leaders indicated that they are highly reliant on their professional associations for both 
social support and research-based resources to help them do their jobs effectively.” Moreover, 
findings from the NCRPP national survey showed that district and school leaders are most likely 
to access research through their professional associations.’ Contrary to many depictions of 
state agencies, state education agency leaders have many social ties across departments and 
with external organizations, ties along which research flows." Research use among state 
education agencies, however, remains a largely understudied phenomenon. 


The Council of State Science Supervisors (CSSS) was the focal professional organization for this, 
study. CSSS is a professional organization composed of science education specialists who serve 
at the state, territorial, or the protectorate educational agency in the United States and U.S. 
Territories. Its members include all current supervisors of science education in State Education 
Agencies, as well as former leaders of their state agency. Within their own jurisdictions, each of 
these supervisors plays a key role in directing efforts at improving school science and to ensure 
excellence and equity in science education. As a professional organization, CSSS organizes 
meetings, facilitates consultations between state leaders, provides learning opportunities to its 
members, and serves as a linkage point from outside organizations to schools and districts. Its 
members are knowledgeable about their states’ standards, curriculum, assessment systems, 
and professional development providers, as well as key state-level initiatives to improve science 
education. 

One such initiative is the recent implementation of the Framework for K-12 Science Education," 
which calls for significant changes to science education to ensure that all students become 
proficient in science. It calls for systems to be organized around building understanding of 
disciplinary core ideas over time, engagement of students in the practices of science and 
engineering, and application of crosscutting concepts that unify science. It is based on a large 
body of research on how students best learn science“ and on careful observations of the real 
work of scientists and engineers.” In the past two years, a number of states have chosen to. 
adopt the Next Generation Science Standards* or standards based on the Framework, an 
important first step to reorganizing science education to achieve the ambitious aims laid out in 
the Framework. Additional changes to curriculum, instruction, teacher preparation and 
professional development, and student assessment will be required. 


In this context, we sought to develop an understanding of CSS members’ work, and their use 
of research to inform their state’s decisions. We asked: 
+ What roles and activities do CSSS members take on and participate in as state science 
education leaders? 
+ What research findings and research-based resources do CSSS members use to inform 
their efforts to promote implementation of the Framework? What research findings and 
research-based resources do they share with other science education leaders? 


+ To whom do state science education leaders turn for research to inform their state’s 
decisions regarding implementation of the Framework? 


In this report, we provide information related to the study design and survey sample, as well as 
the roles and activities in which survey respondents reported engaging as CSSS members and 
state science supervisors. We also present findings related to CSSS members’ research use, with 
respect to the sources and individuals they sought out for research to inform their state’s 
decisions, as well as their efforts to acquire and use research. In future work, we will investigate 
the relationships between CSSS members’ roles and activities and their uses of research 


Study Design 


We surveyed a nationwide sample of CSSS members using items developed through previous 
NCRPP pilot studies and in collaboration with CSSS members. Below, we describe the 
population targeted, procedures for data collection, the content of the survey items, and the 
final sample achieved. 


Population 

The population for our study is the membership of the Council of State Science Supervisors, 
which is comprised of current and former state leaders in science education. Some of these 
members are or were employees of their state agency, while others are or were consultants, 
These include state science supervisors and, in some larger states, their state-level staff. The 
reason to limit the sample in this fashion is to focus on research use that is consequential in 
current state-level policy making in science. We developed a list of current and former CSSS 
members from the organization's website, http://www.csss-science.org/members/, which 
included a total of 98 individuals across 47 states. Former members were listed as Honorary or 
Associate Members. 


Data Collection Procedures 

We collected surveys from current and former CSSS members in two ways. First, we attended 
the 2016 Council of State Science Supervisors Annual Conference, and asked participants to 
complete a paper or online survey during the CSSS General Meeting. Second, we sent email 
invitations to any individuals listed as current or former CSSS members on the membership 
website who had not yet completed the survey. A maximum of three follow-up invitations were 
sent over a one-month period. All respondents opted to complete the survey via Qualtrics, an 
online survey administration platform. On the basis of Qualtrics data, the average respondent 
spent about 22 minutes completing the survey. 


Survey Instrument 
For each survey construct, we provide a definition, sample items, item response choices, and 
the total number of items for that construct. The survey instrument is available by request. 


Research CSSS members found useful. Following an approach used in the national 
NCRPP survey, we sought to identify individual pieces of research that CSSS members found 
Useful in informing a decision in their state related to the implementation of the Framework. 
For each study identified as useful, we asked them to identify (if they could) the title, author, 
year published, publisher, topic, and why they found it useful. 


Research CSSS members shared. We also sought to identify individual pieces of 
research that CSSS members shared with state or district leaders related to their 
implementation of the Framework. For each study identified as useful, we asked them to 
identify (if they could) the title, author, year published, publisher topic, and with whom they 
shared it. 


Acquisition effort. This construct refers to the extent to which an individual exerts effort 
to acquire research and develop relationships with researchers in order to address problems or 
decisions. The scale for acquisition effort was comprised of five items, which asked CSSS 
members to indicate how often they engaged in activities such as contacting researchers to find 
out more about articles they have written, or when confronted with a new problem or decision. 
Item response choices were: Never (1), Rarely (2), Sometimes (3), Often (4), All of the time (5). 


Sources where CSSS members obtain research. These items pertain to the sources 
where CSSS members obtain research, and the frequency with which they consult those 
sources. Sources included traditional ones, such as university researchers, as well as peer 
networks (e.g., professional associations such as the National Science Teachers Association) and 
media. For each source, we asked how often individuals had sought out or acquired research in 
the past twelve months. Item response choices were: Never (1), Rarely (2), Sometimes (3), 
Often (4), All of the time (5). 


CSSS activities that support research use. These items asked participants to self-report 
the frequency with which they have participated in association activities in the past three years, 
as well as the perceived value of those activities for identifying research and research-based 
tools. A list of activities was generated from a review of documents provided by the CSSS Board, 
and then reviewed by Board members. Examples of activities include: annual CSSS meeting 
participation, consultations with other CSSS members, participation in CSSS-sponsored 
webinars, and attendance at workshops/talks by researchers. For each activity, respondents 
were asked how often they participated in the activity, with response choices as follows: Never 
(2), Once (2), 2-3 times (3), 4 or more times (4). Then, for those activities in which respondents 
participated, they were asked to indicate whether or not they: 1) learned about research 
findings, 2) learned about research-based tools, and/or 3) learned about strategies for 
addressing issues in their state. 


Research networks. These items sought to identify to whom CSSS members turned in 
the past 12 months for research to inform their state’s efforts to implement the Framework. 
First, participants were asked to name up to 12 researchers, state science supervisors, and 
other colleagues, both within and outside their states. For each person listed, participants were 
then asked to indicate how often they turned to each person over the last year, considering all 
forms of communication such as face-to-face, e-mail, or telephone, with the following response 
options: 1-2 times per year (1), 3-4 times per year (2), Every 2 months or so (3), Monthly (4), 2-3 
times per month (5), and weekly (6). Then, participants were asked what topics of research they 
discussed with each person among the following five options: curriculum, assessment, 
professional development, instruction, other. 


State science leader activities. These items pertain to the activities participants engaged 
in as part of their work as state science supervisors. In consultation with CSSS board members, 
we developed a list of such activities across six broad areas: standards and curriculum (e. 
reviewing state science standards), assessments (e.g., designing state assessments), 
professional development (e.g., writing contracts for professional development providers), 


partnerships (e.g., identifying resources to share with districts), awards (e.g., coordinating 
student scholarships), and other. Participants were asked to indicate how often they engaged in 
each activity in the last 12 months. Item response choices for each activity were: Never (1), 
Rarely (2), Sometimes (3), Often (4), All the time (5). 


CSSS roles. Roles refer to the roles participants have taken on as CSSS members. We 
developed this list of roles in consultation with CSSS board members. Altogether, we identified 
nine different roles that CSSS members could choose from: 

+ President 

+ Secretary 

+ Board member 

+ Ad-Hoc Committee Chair 

+ Ad-Hoc Committee Contributor/Participant 

+ Presenter at this year’s CSSS conference 

+ Presenter at a previous CSSS conference 

+ Participant in state science conference 

+ Organizer of state science fair 
Respondents could choose more than one role, and indicate “other” if necessary. When 
possible, we re-coded other into one of the nine categories. 


Sample 

Of the 98 individuals invited to complete the survey, 61 of those individuals filled out all or a 
portion of the survey, for an overall response rate of 62%. Of those 61 individuals, 52 
completed the entire survey, for a completion rate of 85%. Of the 47 states in the study 
population, 38 states, or 81%, are represented in the sample. The vast majority of respondents 
(87% or n=53) were current CSS members, while 13% or 8 respondents were former (i.e., 
Associate or Honorary) CSSS members. 


While survey respondents reported holding a variety of positions in their states, nearly half 
(44% or n=27) reported serving as Science Specialists, Coordinators, or Directors (see Figure 1). 
Others reported holding positions as STEM Directors (12% or n=7), Program Managers (10% or 
), science or general educational consultants (12% or n=7). Six respondents, or 10% of the 
sample, indicated that their positions focused on science assessment. 


Education 
Other, 5, 8% _ Associate, 3, 5% 


Educational 
Consultant, 2, 3% _ 


Science Consultant , Science Specialist, 
5, 8% Coordinator, or 
_ Director, 27, 44% 


Science 
Assessment, 6, 10% _ 


Science (& Math) 
Program Manager, 
6, 105% STEM Director, 7, 

12% 


Figure 1. Job Positions of Survey Respondents (n=61) 


On average, respondents reported five years of experience as state science leaders, with a 
range of 0 to 18 years. While 20% (n=12) had one or fewer years of experience, 15% (n=9) had 
more than 10 years of experience. In terms of education, 4% (n=2) reported completing an 
Associate's degree, 63% (n=38) a Bachelor’s degree, 85% (n= 52) a Master's degree, and 33% 
(n=20) have completed or are in the process of completing a doctoral degree. 


Additionally, 38% (n= 23) reported holding a general teaching certification, 77% (n=47) a 
science teaching certification, and 40% (n=24) an administrator certification. Only 6%, or 3 
respondents, indicated not holding any type of certification. 


With respect to gender and race, 69% (n=42) of respondents are female, and 81% (n=50) 
indicated that they are white, with 4% (n=2) African American or Black, 2% (n=1) American 
Indian or Alaska Native, 2% (n=1) Asian, and 10% (n=6) two or more races. 
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SSS Member Roles and Activities 


Key Findings: 

> Survey respondents reported frequently accessing information from the CSSS listserv, as 
well as consulting or collaborating with other states. 

> Survey respondents were most likely to be involved in state-level activities focused on 
reviewing or developing state science standards, designing state science assessments, and 
designing or conducting professional development. 


Roles CSSS Members Assume 

Figure 2 shows CSSS members’ responses to the question regarding the roles they have 
assumed within the organization. Respondents could select multiple roles. The majority of CSSS 
members (67% or n=41) indicated that they have participated in their state's science 
conference. About one-third of respondents (33% or n=20) have served as ad-hoc committee 
contributors or participants, and about one-quarter have served as an ad-hoc committee chair 
(23% or n=14) or a presenter at a CSSS conference (21% or n=13 at a previous conference, and 
18% or n=11 at this year’s conference). Less frequently assumed roles among respondents 
included organizer of state science fair (7% or n=4), as well as CSSS President (7% or n=4) and 
CSSS Secretary (2% or n=1). 


Participant instate scence conference I 7, 
Ad-Hoc Committee Contebuter/Particpant TTT ao, 
AdHoc Committee Chair NNN 2276 
Presenter at aprevius CSSS conference TT 21% 
Presenter at this year's SSS conference [NN 12% 
Board member I 15% 


Organizerof statescience fair [NN 7% 


President (EIN 7% 


secretary [2% 


O% 0% KOKA 
Figure 2. CSSS Roles Taken on by Survey Respondents (n=61) 
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CSSS Activities Members Participate In 

In addition to asking about the roles within CSSS that members have assumed, we asked about 
the types of CSSS activities they have engaged in over the last three years, and the frequency at 
which they participated. Of the 51 respondents who replied to this set of questions, a full 86% 
(n=44) indicated reading information from the CSSS listserv four or more times in the last three 
years (see Figure 3). Additionally, about half of respondents reported consulting with other 
CSSS members at that frequency (51% or n=26), as well as attending workshops or talks by 
researchers (48% or n=24) and collaborating with other states (46% or n=23). About one-third 
of respondents also reported participating in CSSS-sponsored webinars (38% or n=19) and CSSS 
committee meetings (35% or n=17) four or more times, as well as reviewing drafts of the Next 
Generation Science Standards (33% or n=16). CSSS activities that members reported rarely 
engaging in included CSSS Board Meetings, presenting at the National Research Council (NRC) 
or other national meetings, and visiting other states, with 78% (n=40), 57% (n=29), and 62% 
(n=32) reporting no participation in these activities over the last three years, respectively. 


In the last three years, about how often have you participated in each CSSS activity? 
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Figure 3. CSSS Activities in the Last Three Years (n=51) 
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State-Level Activities CSSS Members Participate In 

A total of 52 respondents reported their frequency of participation in various state-level 
activities over the 12 months. These activities centered on five areas: (1) standards and 
curriculum, (2) assessment, (3) professional development, (4) partnerships, and (5) awards. 


Of the activities focused on standards and curriculum, CSSS members reported the most 
frequent engagement in reviewing or developing state science standards, with 61% (n=32) 
serving in lead roles often or all the time, and 46% (n=24) serving in support roles often or all 
the time (see Figure 4a). On the other hand, 48% of respondents (n=25) reported never 
consulting with curriculum companies on products in the design phase, and a full 71% of 
respondents (n=37) reported never organizing state curriculum adoption in the past 12 months. 


Inthe past 12 months, how often have you engaged in the following activities related to 
standards and curriculum? 


Reviewing or developing state sciencestandards(n lead role} 2 


‘Advising state on coutsecontent and/or gad polices (insupport role) 20 


‘Aduisingstate on coursecontent and/or rad polices a ead ole) 2 
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‘rganiangstate curriculum adoption n 


‘Consulting with curriculum companieson productsin the design phase ae 7 EE 


rect 0 10 20 30 40 so 60 70 @ 90 100 


Never Rarely ‘Sometimes soften Allo the time 


Figure 4a. State-Level Activities, Standards and Curriculum (n=52) 
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With respect to state-level activities related to assessment, CSSS members were more likely to 
report more frequent involvement in designing state assessments than selecting contractors for 
state assessments (see Figure 4b). Specifically, 49% of respondents (n=25) indicated that they 
played a support role in designing state assessments often or all the time, and 41% of 
respondents (n=21) reported playing a lead role in the past 12 months. Conversely, only 16% of 
respondents (n=8) reported selecting contractors for state assessments at that frequency. 


Inthe past 12 months, how often have you engagedin the fallawing activities relatedto 
assessment? 


fener sec a - > SE 


Selecting contractors for state assessments a 


Percent 0 -10=S 20S] 00100 


Never Rarely Sometimes soften mAllof the time 


Figure 4b, State-Level Activities, Assessment (n=52) 
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The state-level professional development activities that CSSS members were most frequently 
involved in focused on designing or conducting professional development (see Figure 4c). A full 
73% of respondents (n=38) indicated that they led the design of professional development 
often or all the time in the last 12 months, and 60% (n=31) reported playing a support role in 
professional development design at that frequency. With respect to conducting professional 
development, 65% of respondents (n=34) reported engaging in this type of activity often or all 
the time. On the other hand, CSSS members were much less likely to report writing contracts 
for professional development providers or allocating Title 2A funds at that frequency, with just 
23% (n=12) and 12% (n=6) reporting involvement in these activities often or all the time. 


Inthe past 12 months, haw often have you engagedin the following activities relatedto 
professional development? 


conducting PO 


Designing PD (in support roe) 


Allocating Title 2A funds 32 


Wiring contract for PD providers to suopart state initiatives » » 
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Figure 4c. State-Level Activities, Professional Development (n=52) 
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Compared to the other areas, CSSS members did not report much variation in their level of 
participation in the various partnership activities included on the survey (see Figure 4d). While 
62% of respondents (n=32) reported identifying resources to share with districts often or all the 
time in the last 12 months, 42% (n=22) reported screening or reviewing such resources at that 
frequency. Additionally, 56% of respondents (n=29) reported establishing partnerships with 
business, industry, and non-formal groups often or all the time, and 51% (n=27) reported 
collaborating with their state university systems in K-16 initiatives at that frequency. 


In the past 12 months, how often have you engagedin the following activities relatedto 
partnerships? 


Poneenen ae > || Em 


Collaborating with yourstateuniversty system ink-1Sinittves 8 


Sevanngorcving resources sare wth its (7 /_ Ea 
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Figure 4d. State-Level Activities, Partnerships (n=52) 
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The final area of activities focused on awards (see Figure 4e). In general, respondents were not 
as frequently engaged in state-level award activities as some of the other activities noted above 
(e.g., reviewing or developing state standards or designing or conducting professional 
development). Of the award activities we asked about on the survey, 40% of respondents 
(n=21) reported coordinating teacher awards often or all the time over the last 12 months, 
compared to 31% (n=16) conducting grant competitions and 30% (n=15) coordinating student 
scholarships at that frequency. 


In the past 12 months, how often have you engagedin the following activities relatedto 
awards? 


Coordinating teacherawards (2g, PAEMST, Teacher ofthe Year) ES) 


Coordinating student scholarships (eg, NYSC, state scholarships) a 
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Figure 4e. State-Level Activities, Awards (n=52) 
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Research Use 


Key Findings: 

> The research CSSS members used or shared most often focused on instructional practices 
and student learning in science. 

> Whereas CSSS members used research in a variety of forms, including research or policy 
reports, journal articles, books, they tended to share research in the form of reports. One 
of the most prominent forms of research used and shared were reports by the National 
Research Council. 

> Few, if any, pieces of research that CSSS members used or shared focused on particular 
student groups (e.g., by race/ethnicity, gender, socioeconomic status, or language). 

> Many respondents reported using research to inform their own professional learning, or 
the learning of others. They reported sharing research with a variety of leaders, at the 
state, district, and school levels. 

> CSSS members tended to access research through their professional networks, including 
through CSSS, NSTA, and colleagues in state departments of education. University 
researchers were also prominent sources of research. 

> SSS members turn to a variety of individuals for research to inform their state’s decision, 
including current and former CSSS members, university researchers, state department of 
education leaders, individuals working for educational non-profit organizations, and local 
educational leaders. Of these groups, former CSSS members were particularly prominent 
in CSSS members’ research networks. 


Analysis of Research CSSS Members Use and Share 

We asked respondents two questions related to the specific pieces of research they refer to in 
their work. First, we asked them to name a piece of research they have used to inform their 
state’s decisions related to implementation of the Framework for K-12 Science Education. 
Second, we asked respondents to name a piece of research they have shared with state or 
district leaders related to implementation of the Framework. 


For the first item related to a specific piece of research used to inform state decisions, 27 
unique pieces of research were listed by a total of 38 respondents, or 62% of the overall 
sample. Within these 27 responses, 26 provided enough information to identify the specific 
piece of research referenced. For the second item related to a piece of research shared with 
state or district leaders, 27 unique pieces of research were listed by 33 respondents, or 54% of 
the sample. Within these responses, 24 provided enough information to identify the specific 
piece of research referenced. 


Below, we describe the focal topics of the research named as well as the content areas and 


student subgroups the pieces of research highlighted. Then, we describe the form of the 
research referenced by respondents and the reasons it was useful or with whom it was shared. 
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Focal topics of research. Of the 26 unique pieces of identifiable research listed by 
respondents as useful in informing state decisions, the majority (62% or n=16) focused on 
teachers and teaching in the classroom (see Figure 5). Among these pieces of research, 10, or 
63% of this category, focused on instructional practices, including the National Research Council 
report, Ready, Set, Science!” Others focused on teacher professional learning (n=3 or 19% of 
this category) or curriculum and standards (2 or 13%). 


Teachers and teaching in the classroom was also a prominent category among the 24 unique 
pieces of identifiable research listed by respondents as shared with state or district leaders. 
Specifically, 50%, or 12 pieces of research, focused on this area (see Figure 5). Among those, 10, 
or 83%, focused on instructional practices, such as the Talk Science Primer.”" Other pieces of 
research in this category focused on teacher professional learning or teacher preparation (n=2 
or 17%). 


Research on student learning and student outcomes was another area that respondents noted 
was useful in informing state decisions related to the Framework, with 10 pieces of research (or 
38%) focused in this area (see Figure 5). Within this category, the majority (n=7 or 70%) focused 
on learning and development (e.g., how students learn science), such as the edited volume, 
Working with Big Ideas." The remainder (n=3 or 30%) focused on student achievement or 
academic outcomes. 


Results were similar for the pieces of research respondents reported sharing with state or 
district leaders, with 11 pieces of research (or 46%) focused on student learning and student 
outcomes (see Figure 5). Within this area, nearly all (n=10 or 91%) focused on learning and 
development, such as the National Research Council report, A Framework for K-12 Science 
Education,*" while the one remaining piece of research focused on student achievement 
outcomes. 


A few of the pieces of research named by respondents also focused on assessment. Of those 
named as useful in informing state decisions related to the Framework, 2 pieces of research, or 
8%, focused on either classroom assessment or standardized testing; this figure was 4, or 17%, 
for those pieces respondents shared with state or district leaders (see Figure 5). An example of 
a piece of research focused on assessment is the National Research Council report, Developing 
Assessments for the Next Generation Science Standards." 


Finally, a few respondents named pieces of research related to school or system improvement 
(n=4 or 15%) as something they used to inform their state’s decisions, as well as something 
they shared with state or district leaders (n=4 or 17%), such as the National Research Council 
report, Guide to Implementing the Next Generation Science Standards." 
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ss Teachers and teaching in the classroom ‘= Student leaning and outcomes 


se Assezement ‘School or system improvement 


Figure 5. Focal Topics of Research Used (n=26) and Shared (n=24) 
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Content areas and student subgroups. Perhaps unsurprisingly, the vast majority of the 
pieces of research named by respondents as informing their state’s decisions related to the 
Framework focused on the content area of science (n=23 or 88%). The other 3 studies did not 
focus on a particular content area; instead, they focused generally on assessment development 
or the use of curriculum materials. The same was true for the pieces of research respondents 
reported sharing with state or district leaders, as nearly all (n=23 or 96%) focused on the 
content area of science, with only one piece of research focusing on mathematics. 


With respect to student subgroups, respondents overwhelming named research that did not 
focus on a particular subgroup (n=23 or 88% of those used, and n=22 or 92% of those shared). 
Nonetheless, three pieces of research respondents named as useful in informing their state’s 
decisions focused either on several subgroups (i.e., race/ethnicity, SES/poverty) or specifically 
on English language learners. 


Form of research. The pieces of research that respondents used to inform their state’s 
decisions came in a variety of formats (see Figure 6), including research reports or policy briefs 
(n=9 or 35%), peer-reviewed journal articles (n=7 or 27%), books (n=5 or 19%), presentations 
(n=2 or 8%), research-based tools (n=1 or 4%), and webinars (n=1 or 4%). Six of the research 
reports named, or 23%, were book-length reports published by the National Academies Press 
with support from the National Research Council, such as Ready, Set, Science?” and Developing 
Assessments for the Next Generation Science Standards." 


With respect to the pieces of research shared with state or district leaders, the majority were 
research reports or policy briefs (n=13 or 54%) (see Figure 6), with 8 (33%) published by the 
National Academies Press with support from the National Research Council. Of the other pieces 
of research respondents shared, 4 or 17% were peer-reviewed journal articles, 3 or 13% were 
research-based tools, 2 or 8% were books, and 2 or 8% were presentations. 
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Figure 6. Form of Research Used (n=26) and Shared (n=24) 
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Reasons for why a piece of research was useful. For the item in which respondents 
listed a piece of research they use to inform their state’s decisions related to implementation of 
the Framework, we also asked respondents to answer the question, “Why was it useful?” 
Responses from 28 respondents provided enough information to identify reasons for the 
usefulness of the research. An additional 10 respondents provided no response or an answer 
that provided a description of the piece (e.g., “Study of formative assessment practices”) rather 
than a reason. These responses were excluded from the analysis. 


Respondents’ reasons for using a piece of research to inform their state’s decisions related to 
supporting their own professional learning (n=13 or 46%); providing instructional leadership for 
others in their states (n=6 or 21%); designing programs or initiatives (n=7 or 25%); persuading 
others on programs or initiatives (n=2 or 7%), and supporting and monitoring implementation 
(n=1 or 4%). 


Groups with whom research was shared. For the item in which respondents listed a 
piece of research they shared with state or district leaders, we also asked respondents to 
answer the question, “With whom did you share it?” Responses from 28 respondents provided 
enough information to identify with whom they shared it. An additional 4 respondents provided 
no response or an answer that provided a description of the piece (e.g., “Study of formative 
assessment practices”) or a reason they use it, rather than a group with whom they share it. 
These responses were excluded from the analysis. 


Respondents reported sharing research with individuals at a variety of levels, including other 
state leaders (n=9 or 32%), district leaders (n=7 or 25%), school leaders (n=5 or 18%), teacher 
leaders (n=7 or 25%), and teachers (n=7 or 25%). Three respondents, or 11%, also indicated that 
they posted the piece of research on their state’s website. 
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Sources CSSS Members Use to Obtain Research 

Aset of survey questions asked respondents to report how often they had sought out or 
acquired research from various sources in the past 12 months (see Figure 7). Respondents 
showed a tendency to access research through their networks, such as via their colleagues in 
state department(s) of education (n=35 or 63% often or all the time), the National Science 
Teachers Association (NSTA) (n=31 or 56% often or all the time), and the Council of State 
Science Supervisors (n=31 or 55% often or all the time). Additionally, university researchers and 
conferences represent other prevalent sources for accessing research, with 26 and 23 
respondents, or 48% and 42%, respectively, reporting seeking out or acquiring research from 
these sources often or all the time. 


On the other hand, respondents were less likely to report accessing resources at that frequency 
from the National Center for Education Statistics (n=6 or 11% often or all the time), the What, 
‘Works Clearinghouse (n=6 or 10% often or all the time), Regional Educational Laboratories (n=5 
or 9% often or all the time), vendors (n=4 or 8% often or all the time), the National Science 
Education Leadership Association (n=3 or 6% often or all the time), and county offices of 
education (n=2 or 4% often or all the time). In fact, a majority of respondents indicated that 
they rarely or never acquired research from these six sources. 


During the past 12 months, how often have you sought aut or acquired research from each source? 


ther colleaguesin state departments) of education i i 
National Science Teacher Asscltion (NSA) 
CounclofstateSelenceSuperdsors(¢SS) 2 
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consultants workngwith the tate 
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Figure 7. Sources CSSS Members Use to Access Research (n=55) 
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Effort to Acquire Research 

We asked respondents five questions related to their efforts to acquire research and develop 
relationships with researchers, when confronted with new problems or decisions. The vast 
majority of respondents (n=46 or 75%) indicated that they find it valuable to consult 
educational research “often” or “all of the time” when confronted with a new problem or 
decision (see Figure 8). Additionally, the vast majority of respondents (n=46 or 76%) reported 
looking for relevant research studies “often” or “all of the time” when confronted with a new 
problem or decision. Although 42% of respondents (n=26) reported that they contact 
researchers they already know “often” or “all of the time,” few reported reaching out to 
researchers to find out more about a specific article (n=11 or 18%) or contacting researchers 
they do not know (n=3 or 5%) that frequently. 


When confronted witha new problem or decision, find it 
‘valuable to consult education research, 


research studies that might be relevant. 


When confronted witha new problem or decision, I contact 


a a | 


‘When confronted witha new problem or decision, contact a 
researchers! don't already know who have relevant expertise 


contact researchers to fr in ‘about articles they have ey 7 EE 


Never a Rarely Sometimes often I Allof the time 


Figure 8. Effort to Acquire Research (n=61) 
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CSSS Activities that Research Supports 

Research findings and research-based tools supported the work of state science leaders to 
varying degrees across the activities in which they engaged as CSSS members. The vast majority 
of respondents who reported attending the CSSS annual meeting or BCSSE meetings, as well as, 
‘workshops or talks by researchers, also reported learning about research findings and research- 
based tools through these activities (see Figure 9). About two-thirds of respondents who 
reported reading information from the CSSS listserv, participating in CSSS-sponsored webinars, 
and attending CSSS committee meeting also reported learning about research findings and 
research-based tools through these activities. Conversely, fewer respondents who reported 
visiting other states or attending CSSS Board meetings indicated that they learned about 
research findings or research-based tools during those activities. 


If you participated in the activity, in what ways did it support your work? 
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Figure 9. CSSS Activities that Research Supports (16 < n= 49) 
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Research Networks 

Survey questions pertaining to research networks asked respondents to name up to 12 
individuals to whom they have turned in the past 12 months for research to inform their state’s 
efforts to implement the vision of the Framework for K-12 Science Education. We also asked 
respondents to indicate the topics of research discussed: curriculum, assessment, professional 
development, and/or instruction. 


The 44 CSSS members (37 current, 7 former) who responded to these questions named a total 
of 141 individuals to whom they turned for research. We categorized these individuals into six 
groups, depending on their organizational affiliations: 

* Current CSSS members (n=16) 

* Former CSSS members (n=6) 

+ University researchers (n=75) 

+ State department of education staff (n=8) 

+ Individuals working in education non-profits, such as Achieve, the Exploratorium, 

SRI, and WestEd (n=29) 
+ Local educational leaders, such as school district administrators (n=7) 


‘We analyzed these data in several ways. First, we calculated the number of individuals to whom 
each respondent turned for research, both overall and by research topic. In social network 
analysis, this number is referred to as an individual's out-degree. Those with higher out-degrees 
in this network can be interpreted to seek out more people for research to inform their state’s 
implementation efforts. Second, we calculated the number of individuals who named each 
person in the network as someone they turn to for research, both overall and by topic. This 
number is referred to as an individual's in-degree. Those with higher in-degrees are often 
thought to have more influence in a network. In general, both out-degree and in-degree help us 
to get a sense of an individual's overall exposure to others in the network, and in this case to 
research and research-related information. We report average out-degree and in-degree 
below, by research focus area and organizational affiliation 


The 44 CSSS members who responded to the network questions had an average out-degree of 
six, indicating that they turned to an average of six individuals for research to inform their 
state’s efforts to implement the vision of the Framework (see Table 1). Across the focal areas, 
respondents reported turning to an average of 3 individuals for research related to assessment 
and curriculum, and an average of 4 individuals for research related to instruction and 
professional development (see far left column in Table 1). Current CSSS members reported 
turning to at least one more individual for research related to each topic area than former CSSS 
members, indicating that current members are more slightly more active in seeking out 
research related to their state’s efforts than former members. 
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Table 1. Average (SD) Out-Degree, or Number of Individuals to Whom Respondents 
Turned for Research Related to Different Topics 


Research Topic All Current Member Former Member 
i (n=44) (n=37) (n=7) 
6.2 64 5.1 
a (3.8) (3.7) (4.3) 
Assessment 28 3.0 20 
(2.9) (2.9) (2.8) 
Curriculum 33 35 23 
(2.6) (2.7) (2.0) 
Instruction 40. 43 24 
(3.1) (3.1) (2.6) 
Professional 4.0 41 33 


Development (2.9) (3.0) (25) 


‘Among the 141 individuals who were named as someone a respondent turned to for research 
to inform their state’s efforts to implement the Framework, each had an average in-degree of 
two, indicating that they were named an average of two times by respondents (see Table 2). 
There was not much variation in in-degree across research areas, with in-degrees ranging from 
an average of 0.9 for assessment and 1.0 for curriculum, to 1.2 for instruction and professional 
development. 


Comparing in-degree across organizational affiliation, it is notable that former CSSS members 
were named most frequently as someone respondents turn to for research, by an average of 7 
respondents (see Table 2). Even so, there is wide variability in how often these individuals were 
named, as evidenced by the standard deviation of 6.8, indicating that just a few former CSSS 
members were named by several respondents. Thus, while former CSSS members may not be 
as active in seeking out research as current CSSS members (see Table 1), a few key former CSSS. 
members are prominent sources of research in the network. 


Of the other groups, current CSSS members, university researchers, and individuals from non- 
profit organizations were all named by an average of two respondents, with average in-degrees 
of 1.8, 1.9, and 1.6, respectively. State department of education staff and local leaders were 
both named by an average of one respondent. 


Table 2. Average (SD) In-Degree, or Number of Times Respondents Named Individuals as Someone They 
Turn to for Research 


Current Former —_ University Local 
bees id 1) Member Member researchers Seen ena leader 
i ‘ (n=16) (n=6) (n=75) = : (n=7) 
All 19 18 6.38 ig 1.0 16 10 
(2.5) (1.2) (6.8) (2.4) (0.0) (1.3) (0.0) 
‘Assacsmant 09 e, 4.0 08 04 0.6 0.6 
(1.6) (0.6) (4.4) (1.5) (0.5) (0.7) (0.5) 
Curriculum 1.0 08 3.5 1.0 08 1.0 07 
(1.6) (0.9) (4.2) (1.6) (0.5) (1.1) (0.5) 
Instruction i2 1.0 4.2 i2: 0.6 11 10 
(1.8) (14) (5.7) (1.5) (0.5) (1.3) (0.0) 
Professional a aS. ae: ake 05 it 0.9 
Development (2.0) (1.4) (6.2) (1.6) (0.5) (0.9) (0.4) 
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Another way to examine the CSSS members’ research network is through a visual 
representation of the network, called a sociogram. In addition to generating sociograms for the 
overall network and by focal area, we calculated two network-level measures to help us 
understand the levels of cohesion and connectedness in each network. The first measure is 
density, which is simply the number of interactions in the network, or ties, expressed as a 
proportion of the number of possible ties. Density is an indicator of cohesion, and is best 
interpreted when compared between networks, or between focal areas in our case. The second 
measure is connectedness, which measures the extent to which a network is connected via 
examining the proportion of pairs of individuals, or nodes, in the network who can reach each 
other via others in the network. 


We first present the overall research network sociogram in Figure 10a, where each dot, or 
node, indicates a unique individual in the network. Organizational affiliations are indicated by 
color, as noted in the key. We also present the same network with nodes sized by in-degree 
(see Figure 10b}, which helps to get a sense of which individuals are most prominent in the 
network. 


‘An examination of Figure 10a reveals that CSSS members have a relatively well-connected 
research network, given that the vast majority of individuals in the network are connected to at 
least one other person, and many individuals are connected to the same people. That is, there 
is some overlap in who respondents turned to for research, and very few CSSS members turned 
to someone to whom no other members turned. Indeed, the overall connectedness for the 
research network is 0.819 (see Table 3), indicating that 80% of people can reach each other via 
others in the network. 


Upon sizing the nodes by in-degree (see Figure 9b), where the larger the node indicates a 
higher number of respondents who named that person as someone they turn to for research, 
we see that a handful of individuals emerge as most central or influential in the network. 
Specifically, two former CSSS members (yellow nodes) have among the largest nodes in the 
network, followed by three or four university researchers (turquoise nodes), and two current 
CSSS members (red nodes). 
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Key Red Current CSSS member 
Yellow Former CSSS Member 
Turquoise University-based researcher 


Green _ Non-Profit Organization 
Dark blue State Department of Education 
Pink Local educational leader 


Figure 9a. Respondents’ Research Network Figure 9b. Research Network by In-Degree 
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Considering the focal areas around which CSSS members turned to others for research, research 
about instruction and professional development was shared more frequently than research on 
assessment and curriculum. The instruction and professional development research networks were 
slightly denser than the assessment and curriculum networks (0.006 compared to 0.005, see Table 
3), suggesting that CSSS members’ research networks in instruction and professional development 
are a bit more cohesive. Moreover, while about 40% of individuals in the instruction and 
professional development networks can reach one another via others in the network, this 
proportion was about 30% for the curriculum network, and 20% for the assessment network (see 
connectedness in Table 3). For visual representations of these four networks, see Figures 11a-114. 


Table 3. Network Density and Connectedness 


Network. Density Connectedness 
All 0.010 0.819 
Assessment 0.005 0.226 
Curriculum 0.005 0.315 
Instruction 0.006 0.409 
Professional Development 0.006 0.402 
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Figure 1c. Professional Development Figure 11d. Instruction 
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Key Conclusions and Next Steps 


Findings from this study suggest that CSSS members regularly use research to inform their 
state’s decisions related to implementation of the Framework for K-12 Science Education. They 
tend to access research through their networks, such as CSSS, NSTA, or state department of 
education colleagues, rather than via other research outlets such as governmental websites, 
Moreover, broad engagement in CSSS networks, such as through meetings and workshops, 
appear to facilitate the use of research more than activities like visiting or collaborating with 
other states. Indeed, our findings show that CSSS members have a relatively robust, well- 
connected research network. Within that network, former CSSS members, or Honorary and 
Associate Members, are key sources of research, suggesting that the inclusion of these 
members in the organization’s activities helps to preserve institutional memory around 
research use. 


With respect to the kinds of research that CSSS members found useful and shared with others 
in their states, National Research Council reports published via the National Academies Press 
were particularly prominent, indicating that these research syntheses are important for 
informing the work of state science leaders. Nonetheless, it was striking that very few, if any, of 
the research studies that CSSS members used or shared focused on science education as it 
pertains to particular student subgroups. In the context of changing demographics in states and 
districts across the country, it will be important to consider whether and how science education 
reforms are implemented equitably for students across race, gender, socioeconomics, and 
language, and for research to attend to these issues. 


The next steps in this work are to explore the relationships between CSSS members’ roles and 
activities and their uses of research. This analysis will help us to understand whether particular 
kinds of roles or activities are associated with CSSS members’ access to research, as well as 
their efforts to use research. We will also conduct a second round of survey data collection in 
spring 2017, which will allow us to examine how CSSS members’ research network shifts as new 
leaders join the network. 
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